Papers with systematic empirical analysis
Divide-Conquer-Reasoning for Consistency Evaluation and Automatic Improvement of Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality and consistency of text generated by Large Language Models are not effective. |
| Approach: | They propose a divide-conquer-reasoning approach to evaluate LLM-generated texts using a split-and-conquers evaluator and an automatic metric converter to facilitate this approach. |
| Outcome: | The proposed framework outperforms state-of-the-art methods by a large margin on multiple benchmarks and reduces 90% of output inconsistencies in one iteration. |
Probing Pretrained Language Models for Lexical Semantics (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on morphosyntactic, semantic, and world knowledge, but it remains unclear to what extent LMs derive lexical type-level knowledge from words in context. |
| Approach: | They propose to use multilingual and monolingual LMs to extract lexical type-level knowledge from words in context. |
| Outcome: | The proposed models perform well across six typologically diverse languages and five lexical tasks. |
SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection rely on self-consistency check alone . prominent LMs exhibit a tendency to produce exceedingly confident, but erroneous, assertions . |
| Approach: | They propose a sampling-based method that expands on the principle of self-consistency checking to detect hallucinations at question-level and model-level. |
| Outcome: | The proposed method outperforms the state of the art in detecting non-factual and factual statements across multiple question-answering and open-domain generation benchmarks. |
AgentAsk: Multi-Agent Systems Need to Ask (2026.acl-long)
Copied to clipboard
Bohan Lin, Kuo Yang, Zelin Tan, Yingchuan Lai, Chen Zhang, Guibin Zhang, Xinlei Yu, Miao Yu, Xu Wang, Yudong Zhang, Yang Wang
| Challenge: | Multi-agent systems fail to consistently outperform strong single-a agent baselines due to error propagation at inter-aggent message handoffs. |
| Approach: | They propose an edge-level error taxonomy that identifies four main errors in multi-agent interactions as data gaps, signal corruption, referential drift and capacity gaps as primary sources of failure. |
| Outcome: | The proposed module outperforms existing systems on five benchmarks and is architecture-agnostic. |